In view of the problems that classroom teaching scene is obscured seriously and has numerous students， the current video action recognition algorithm is not suitable for classroom teaching scene， and there is no public dataset of student classroom action， a classroom teaching video library and a student classroom action library were constructed， and a real-time multi-person student classroom action recognition algorithm based on deep spatiotemporal residual convolution neural network was proposed. Firstly， combined with real-time object detection and tracking to get the real-time picture stream of each student， and then the deep spatiotemporal residual convolution neural network was used to learn the spatiotemporal characteristics of each student’s action， so as to realize the real-time recognition of classroom behavior for multiple students in classroom teaching scenes. In addition， an intelligent teaching evaluation model was constructed， and an intelligent teaching evaluation system based on the recognition of students’ classroom actions was designed and implemented， which can help improve the teaching quality and realize the intelligent education. By making experimental comparison and analysis on the classroom teaching video dataset， it is verified that the proposed real-time classroom action recognition model for multiple students in classroom teaching video can achieve high accuracy of 88.5%， and the intelligent teaching evaluation system based on classroom action recognition has also achieved good results in classroom teaching video dataset.